HW-ADAM: FPGA-Based Accelerator for Adaptive Moment Estimation

The selection of the optimizer is critical for convergence in field on-chip training. As one second moment optimizer, adaptive estimation (ADAM) shows a significant advantage compared with non-moment optimizers such as stochastic gradient descent (SGD) and first-moment Momentum. However, ADAM hard to implement on hardware due computationally intensive operations, including square, root extraction, division. This work proposed Hardware-ADAM (HW-ADAM), an efficient fixed-point accelerator highlighting hardware-oriented mathematical optimizations. HW-ADAM has two designs: Efficient-ADAM (E-ADAM) unit reduced resource consumption by around 90% related work. E-ADAM achieved throughput 2.89 MUOP/s (Million Updating Operation per Second), which 2.8× original ADAM. Fast-ADAM (F-ADAM) 91.5% flip-flops, 65.7% look-up tables, 50% DSPs F-ADAM 16.7 MUOP/s, 16.4×